fix(client): force Event Hubs to use V1_0_0_0 #2633
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The problem with supporting Azure Event Hubs is that it isn't Apache Kafka under the covers, it's an intermediate proxy that supports a subset of the Kafka APIs at various versions and then maps them onto Event Hubs protocol(s) at the backend. As as result Sarama's current mechanism of specifying the KAFKA_VERSION to determine what protocol versions to support and use doesn't really work properly with Event Hubs because it supports an unusual set of protocols and even defines minimum versions for ProduceRequest (v3) and FetchRequest (v4). For some reason EventHubs is very behind on FetchRequest and MetadataRequest so the max configuration you can use in Sarama is V1_0_0_0 and the minimum is V0_11_0_0. As we have recently bumped the default Version to V2_1_0_0 and support a wider range of protocol versions, we're more likely to see issues raised by Event Hubs users unless they have already pinned their version correctly. To try and prevent this, attempt to detect use with Event Hubs by inspecting the bootstrap broker addresses and overriding the Version. Contributes-to: #2470 Signed-off-by: Dominic Evans <dominic.evans@uk.ibm.com>
dnwe
added a commit
to dnwe/components-contrib
that referenced
this pull request
Aug 30, 2023
Pulling in `fix(client): force Event Hubs to use V1_0_0_0` by @dnwe in IBM/sarama#2633 This should avoid dapr hitting any issues with Azure Event Hubs users who might not have followed the instructions in dapr/docs#3474 to pin their version number. Signed-off-by: Dominic Evans <dominic.evans@uk.ibm.com>
3 tasks
dnwe
added a commit
to dnwe/components-contrib
that referenced
this pull request
Aug 30, 2023
Pulling in `fix(client): force Event Hubs to use V1_0_0_0` by @dnwe in IBM/sarama#2633 This should avoid dapr hitting any issues with Azure Event Hubs users who might not have followed the instructions in dapr/docs#3474 to pin their version number. Signed-off-by: Dominic Evans <dominic.evans@uk.ibm.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The problem with supporting Azure Event Hubs is that it isn't Apache Kafka under the covers, it's an intermediate proxy that supports a subset of the Kafka APIs at various versions and then maps them onto Event Hubs protocol(s) at the backend. As as result Sarama's current mechanism of specifying the KAFKA_VERSION to determine what protocol versions to support and use doesn't really work properly with Event Hubs because it supports an unusual set of protocols and even defines minimum versions for ProduceRequest (v3) and FetchRequest (v4). For some reason EventHubs is very behind on FetchRequest and MetadataRequest so the max configuration you can use in Sarama is V1_0_0_0 and the minimum is V0_11_0_0.
As we have recently bumped the default Version to V2_1_0_0 and support a wider range of protocol versions, we're more likely to see issues raised by Event Hubs users unless they have already pinned their version correctly. To try and prevent this, attempt to detect use with Event Hubs by inspecting the bootstrap broker addresses and overriding the Version.
Contributes-to: #2470